The meta-analysis: supportive or illuminating?

Many of us use statistical methods like an unsteady person might use a lamppost—more for support (of our preconceived notions) than for illumination. After all, Yogi Bera once said, ‘If I didn’t believe it, I wouldn’t have seen it.’ So it is easy to scoff at some of the flimsy constructs our colleagues have brought forward to ‘illuminate’ important issues that have not been answered by definitive trial data. In no case has the derision reached as high a level as with the meta-analysis, an attempt to compile data from small trials to answer important clinical questions. Though there may be good scientific rationale for such an idea all of us enjoy pointing to numerous examples in which poorly executed meta-analyses led to conclusions that were swept away later by rigorous clinical trial results. The truth is that the meta-analysis, like any statistical method, is susceptible to error when performed incorrectly. And one of the most common mistakes is study selection. Inclusion of poor studies and exclusion of valuable ones can dramatically affect the quality of the analysis to the point of reaching conclusions that are exactly contrary to the truth. A second potential weakness of this methodology is that, even with the compilation of well-executed trials, numbers may still be insufficient to reach satisfactory overall conclusions. The reliability of an observation is improved with a large number of events. If events occur only at a low rate, or if the difference between groups is small, the outcome described may be attributable to the play of chance. Small meta-analyses are susceptible (perhaps even more susceptible) to this kind of statistical error. Enter the meta-analysis of Burgess et al. in which they seek to describe the relative value of interventions for the prevention of atrial fibrillation (AF) that regularly occurs after cardiac surgery. The authors complied articles on the basis of relatively strict criteria and applied standard methodology to extract their conclusions. Were their findings unexpected or surprising? We think not. The most commonly tested interventions, beta-blockers, sotalol, amiodarone, and pacing were effective in preventing AF. Magnesium also had an effect but was confounded by concomitant beta-blocker use. Amiodarone and pacing decreased the length of stay, and only alone reduced strokes. So why is this paper valuable? The answer is simple. Because of the size of the study and the scrupulous culling of the literature to exclude study selection bias, as well as the excellence of the analyses, the results can be considered reliable. The fact that the results of the study are consistent directionally with the best studies in this area provides added assurance. However, lest the reader believe we have reached the end of the rainbow on this subject, these editorialists would like to provide a few important caveats about meta-analyses in general and the present study in particular. First, meta-analyses should never be viewed as anything but hypothesis generating. They are susceptible to many confounders including publication bias that cannot be completely accounted for with any statistical methodology. For these reasons, the thoughtful reader might consider taking away a zero from the P-value generated in a meta-analysis and/or to use a 99% confidence interval for ratios of relative risk. As an example, in the present study, the finding of stroke reduction with amiodarone would be lost, an appropriate adjustment we think because this is the weakest part of the study results. Secondly, statistical significance can be a far cry from what a clinician might find compelling. Most of our colleagues embrace the concept of beta-blocker prophylaxis but few use sotalol. Amiodarone’s uptake has been spotty, and virtually no one uses prophylactic pacing. Why? The clear perception of most clinicians is that post-operative AF in most patients is simply not worth the risk associated with these interventions. Even if one were to decide to treat prophylactically, this meta-analysis illustrates how much heterogeneity exists for important issues such as dose, duration of therapy, and definition of success. Thirdly, the length of stay issue is complex. We have learnt that there are many reasons why patients remain in the hospital after surgery, and that trying to impact this parameter simply by preventing AF may be a false hope. We suspect that re-educating surgeons and cardiologists to the concept that AF can be managed conservatively would be helpful. To convince all constituencies of this might require an AFFIRM-like study in post-operative patients, an idea that needs to be developed and pursued. Fourthly, stroke is the overwhelming morbidity associated with AF. The amortized risk in post-operative patients is